Add --coverage option for local.py, make clang coverage builds use -fcoverage-mapping/-fprofile-instr-generate
#37457
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Review changes with Changed Files
|
|
PR #37457: Size comparison from 8deebeb to 678832f Full report (14 builds for cc13x4_26x4, cc32xx, nrfconnect, qpg, stm32, tizen)
|
andy31415
changed the title
--coverage option for local.py--coverage option for local.py, make clang coverage builds use -fcoverage-mapping/-fprofile-instr-generate
|
PR #37457: Size comparison from 8deebeb to 43f94d6 Full report (45 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, nrfconnect, psoc6, qpg, stm32, telink, tizen)
|
|
This pull request enables building the Python test example apps with the --coverage option. However, while the example apps are now built with coverage instrumentation, they still build the SDK independently and don't use the SDK built during the unit tests. I believe this won't affect the final coverage results because the example app's binary isn't included in the statistics folder used by lgcov. But this is good step forward. |
|
PR #37457: Size comparison from 8deebeb to b839967 Full report (72 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
|
I need to iterate on it. Sample apps also use the SDK built through "third_party/...." so paths do not match. I am not sure how (if even possible) to combine them. Still iterating on "gather results" part. |
This pull request enables building and running Python tests with the --coverage option. Future work will focus on how the build_coverage.sh script can utilize this new functionality to improve coverage results. Lets split the work |
yufengwangca
approved these changes
Feb 7, 2025
|
PR #37457: Size comparison from 8deebeb to a18015d Full report (3 builds for cc32xx, stm32)
|
|
PR #37457: Size comparison from 8deebeb to 4c507c4 Full report (9 builds for cc13x4_26x4, cc32xx, stm32, tizen)
|
|
PR #37457: Size comparison from 8deebeb to 96f1443 Full report (70 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink)
|
|
PR #37457: Size comparison from 8deebeb to 9adddfc Full report (20 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, nrfconnect, qpg, stm32, tizen)
|
|
PR #37457: Size comparison from 8deebeb to 1d15fed Full report (72 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
|
|
should we add connectedhomeip/scripts/build/builders/host.py Lines 606 to 620 in e86768b |
…s or tests that we know will fail
This is a very good catch. Thank you! Updated so I force clang for now. |
|
PR #37457: Size comparison from df5ee33 to b5d22f2 Full report (72 builds for bl602, bl702, bl702l, cc13x4_26x4, cc32xx, cyw30739, efr32, esp32, linux, nrfconnect, nxp, psoc6, qpg, stm32, telink, tizen)
|
|
This ended up having a LOT of deltas for fixes, however they affect local running only so I will merge so we can iterate. Since CI passes, there is a strong assumption that this is better than before, but we could make it even better in a few steps. |
pimpalemahesh
pushed a commit
to pimpalemahesh/connectedhomeip
that referenced
this pull request
Feb 12, 2025
…`-fcoverage-mapping/-fprofile-instr-generate` (project-chip#37457) * Add coverage support flags (but no save yet for the flag) * Minor cleanup, save config options when loading variants, so build flags transcend running * Fix yaml content * Restyled by isort * Switch coverage to clang, ensure out dir for config exists * Slight doc update, allow local.py to use ccache * Start adding filter support and comma separation support to local.py * Fix the runner termination logic * gen-coverage works * Restyle * make linter happy * Make it clearer what errors we ignore, add more error logic to make this run on more machines * Restyle * Add more ignores for cleaner results * Restyle * One more ignore * Better organization, make paths consistent * Fix logic. Merged reports now work! * Restyle * fix stderr logging * fix fail dir creation * Make mismatched FIFO pid much easier to investigate * Add hierarchical view by default - it seems easier for us to narrow down coverage * Add coverage support for yaml * Use exec to execute the sub-program * Fix up arguments in local.py to support propper quoting * For chip tool tests also run chip-tool with coverage support * Fix the path-variance runs of unit test, so I can run unit tests with coverage * Updated comment a bit * Increase timeout for test running to cover slow tests, skip slow tests or tests that we know will fail * Also exclude manual tests by default from local runs * match default exclusions ... also flaky excluded * Support keep going for yaml tests, add more ignores * Multiprocessing coverage, coverage now works * Add missing assignment * Restyled by autopep8 --------- Co-authored-by: Andrei Litvin <[email protected]> Co-authored-by: Restyled.io <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This allows coverage option to be passed in to the
local.pyscript as a persistent option (once you build with coverage, tests start running with coverage).Changes:
--coverage/--no-coverageas command line options to local.py-fprofile-generateand-fcoverage-mappinginstead of--coveragesince otherwise I get errors regarding lcov version. Since we pull pigweed clang, we also pull compatible llvm-cov binaries so using that seems to be more consistent.Testing
Ran
buildlocally, observed coverage flag was used in outputs. Ranpython-testsand observed coverage settings were picked up.Ran coverage for both pyhton tests and yaml tests, saw that coverage increases the more tests I run.
Tested that
linux-x64-tests-coveragestill generates coverage HTML